
Calhoun 

iniQiuiic^iul Ar{hiv« of tilt Mil vdl Poii^roduiit School 


Calhoun: The NPS Institutional Archive 
□Space Repository 



Theses and Dissertations 


1. Thesis and Dissertation Collection, all items 


2014-12 

Too good to be used: analyzing utilization of 
the test program for certain commercial items 
in the Air Force 

Fleharty, Matthew J.; Sharkey, John J., Jr. 

Monterey, California: Naval Postgraduate School 
http://hdl.handle.net/10945/44561 

This publication is a work of the U.S. Government as defined in Title 17, United 
States Code, Section 101. Copyright protection is not available for this work in the 
United States. 

Downloaded from NPS Archive: Calhoun 



DUDLEY 

KNOX 

LIBRARY 


htt p://w ww. n ps. e du/l ib ra ry 


Callwuo is the Naval Postgraduate School's public access digital repository for 
research mate rials and institutiional publicatkios created by the NPS community. 
Calhoun is named for Professor of Mathematics Guy K. CaSiouo, NPS's first 
appointed — and published — schoteily author. 

Dudley Knox Library / Naval Postgraduate School 
411 Dyer Road / 1 University Circle 
Monterey, California USA 93943 







NAVAL 

POSTGRADUATE 

SCHOOL 

MONTEREY, CALIFORNIA 


MBA PROFESSIONAL REPORT 


TOO GOOD TO BE USED: 
ANALYZING UTILIZATION OE THE 
TEST PROGRAM EOR CERTAIN 
COMMERCIAL ITEMS IN THE AIR 
EORCE 


December 2014 

By: Matthew J. Fleharty, 

John J. Sharkey Jr. 


Advisors: Elliott Cory Yoder 
Rene G. Rendon 


Approved for public release; distribution is unlimited 







THIS PAGE INTENTIONALLY LEET BLANK 



REPORT DOCUMENTATION PAGE 

Form Approved 0MB No. 0704-0188 

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, 
searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send 
comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to 
Washington headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 
22202^302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188) Washington DC 20503. 

1. AGENCY USE ONLY (Leave blank) 

2. REPORT DATE 

December 2014 

3. REPORT TYPE AND DATES COVERED 

Master’s Thesis 

4. TITLE AND SUBTITLE 

TOO GOOD TO BE USED: ANALYZING UTILIZATION OF THE TEST 
PROGRAM FOR CERTAIN COMMERCIAL ITEMS IN THE AIR FORCE 

5. EUNDING NUMBERS 

6. AUTHOR(S) Matthew J. Fleharty and John J. Sharkey Jr. 


_ 

7. PEREORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

Naval Postgraduate School 

Monterey, CA 93943-5000 

8. PEREORMING ORGANIZATION 
REPORT NUMBER 

9. SPONSORING /MONITORING AGENCY NAME(S) AND ADDRESS(ES) 

N/A 

10. SPONSORING/MONITORING 
AGENCY REPORT NUMBER 

II. SUPPLEMENTARY NOTES The views expressed in this thesis are those of the author and do not reflect the official policy 
or position of the Department of Defense or the U.S. Government. IRB Protocol number N/A 

I2a. DISTRIBUTION / AVAILABILITY STATEMENT 

Approved for public release; distribution is unlimited 

I2b. DISTRIBUTION CODE 

A 

13. ABSTRACT (maximum 200 words) 






The purpose of this project is to analyze the Air Force’s usage rate of the Federal Acquisition Regulation (FAR) 
Subpart 13.5 Test Program for Certain Commercial Items. FAR 13.500(b) requires contracting officers to maximize 
the use of the test program when practicable. In addition to the FAR mandate, the Government Accountability Office 
(GAO), found it is in the government’s best interest to use the program. According to a GAO report conducted in 
February of 2014, titled Commercial Item Test Program Beneficial, but Actions Needed to Mitigate Potential Risks, 
the program improved contract lead-time and reduced required administration without an increase in overall risk to 
the government. Therefore, underutilization of the program will identify inefficiencies in the procurement process. 
This research seeks to use the Federal Procurement Data System-Next Generation (FPDS-NG) data to identify the 
usage rate of the FAR Subpart 13.5 Test Program for Certain Commercial Items in the Air Force and provide 
potential recommendations to increase and improve the test program’s use. 

14. SUBJECT TERMS 

Federal Acquisition Regulation (FAR) Subpart 13.5, test program, Test Program for Certain 

Commercial Items, Simplified Acquisition Procedures (SAP), Air Force, Federal Acquisition 

15. NUMBER OE 

PAGES 

81 

1 Streamlining Act (FASA), Federal Acquisition Reform Act (FARA) 




16. PRICE CODE 

17. SECURITY 
CLASSIEICATION OE 
REPORT 

Unclassified 

18. SECURITY 

CLASSIEICATION OE THIS 
PAGE 

Unclassified 

19. SECURITY 
CLASSIEICATION OE 
ABSTRACT 

Unclassified 

20. LIMITATION OE 
ABSTRACT 

UU 


NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89) 


Prescribed by ANSI Std. 239-18 


1 




























THIS PAGE INTENTIONALLY LEET BLANK 


11 



Approved for public release; distribution is unlimited 


TOO GOOD TO BE USED: ANALYZING UTILIZATION OF THE TEST 
PROGRAM FOR CERTAIN COMMERCIAL ITEMS IN THE AIR FORCE 


Matthew J. Fleharty, Captain, United States Air Force 
John J. Sharkey Jr., Captain, United States Air Force 


Submitted in partial fulfillment of the 
requirements for the degree of 


MASTER OF BUSINESS ADMINISTRATION 

from the 

NAVAL POSTGRADUATE SCHOOL 
December 2014 


Author: Matthew J. Fleharty 


John J. Sharkey Jr. 


Approved by: Elliott Cory Yoder, Thesis Advisor 


Rene G. Rendon, Second Reader 


William Gates, Dean 

Graduate School of Business and Public Policy 



THIS PAGE INTENTIONALLY LEET BLANK 


IV 



ABSTRACT 


The purpose of this project is to analyze the Air Force’s usage rate of the Federal 
Acquisition Regulation (FAR) Subpart 13.5 Test Program for Certain Commercial Items. 
FAR 13.500(b) requires contracting officers to maximize the use of the test program 
when practicable. In addition to the FAR mandate, the Government Accountability 
Office (GAO), found it is in the government’s best interest to use the program. According 
to a GAO report conducted in February of 2014, titled Commercial Item Test Program 
Beneficial, but Actions Needed to Mitigate Potential Risks, the program improved 
contract lead-time and reduced required administration without an increase in overall risk 
to the government. Therefore, underutilization of the program will identify inefficiencies 
in the procurement process. This research seeks to use the Federal Procurement Data 
System-Next Generation (FPDS-NG) data to identify the usage rate of the FAR Subpart 
13.5 Test Program for Certain Commercial Items in the Air Force and provide potential 
recommendations to increase and improve the test program’s use. 
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I. 


INTRODUCTION 


A. BACKGROUND 

On January 1, 1997, the Federal Acquisition Reform Act (FARA) temporarily 
authorized the application of simplified acquisition procedures for certain commercial 
items up to $5 million. Almost 18 years later, the only substantive change to the program 
is an increased threshold of $6.5 million. Today, the authority for the program still 
remains temporary and is, once again, due to expire on January 1, 2015, unless Congress 
renews the act or makes it permanent. 

The importance of the FAR Subpart 13.5 Test Program for Certain Commercial 
Items program can, in part, be traced back to a different dynamic for investments in 
technology. In a report titled Military R&D and Innovation, David Mowery pointed out 
that since the mid-1980s, industry has outpaced the federal government in the amount of 
capital invested in research and development (R&D) projects (as cited in Gansler, 2014). 
As a result, commercial technologies have become the new standard for what is state of 
the art and are able to fulfill more government requirements. Congressional legislation 
was then passed to establish a clear preference for commercial items, now codified in the 
Federal Acquisition Regulations (FAR, 2014) under FAR Part 12, Acquisition of 
Commercial Items. The increasing relevance of commercial items in government 
acquisitions only bolsters the importance and value of any available procedures, such as 
the test program, used to procure those commercial goods and services. 

Another dynamic worth considering regarding the test program is the changing 
federal acquisition workforce. As the Government Accountability Office (GAO, 2013) 
noted, “The acquisition workforce plays a key role in managing programs and overseeing 
contracts to help agencies get what they need, at the right time, and at a reasonable price” 
(p. 1); however, the government is experiencing unique challenges impacting the 
retention of its current employees. In 2012, the Department of Defense (DOD) 
acquisition workforce “decreased for the first time since FY 2004. The decrease [was] 
due to a higher attrition rate, including retirements, and ... fewer new hires” (Federal 
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Acquisition Institute [FAI], 2013, p. 17). Because of increasing budgetary pressures 
resulting from the public’s support for austerity measures and an ever-expanding number 
of eligible retirees, the acquisition workforce should be expected to become smaller and 
less experienced. Appropriately, these professionals will need the right tools and 
procedures to enhance their productivity. With such concerns in mind, FAR Subpart 13.5 
was established to increase efficiencies and alleviate administrative burdens for 
contracting organizations through expanding the usefulness and applicability of 
simplified acquisition procedures. 

B. RESEARCH OBJECTIVES 

The main objective of this research paper is to examine the Air Force’s usage of 
the FAR Subpart 13.5 Test Program for Certain Commercial Items, specifically from 
February 1, 2013 through May 31, 2014, to determine whether contracting offices are 
employing “the simplified procedures authorized by the test to the maximum extent 
practicable” (FAR 13.500[b]). Furthermore, this research attempts to identify and 
investigate various metrics in an effort to analyze the relative effectiveness of the test 
program procedures. Finally, this report makes recommendations on how to increase the 
usage and effectiveness of the test program. More specifically, this report answers the 
following questions: 

1. What was the Air Force’s usage rate of FAR 13.5 Test Program for 
Certain Commercial Items procedures from February 1, 2013, through 
May 31,2014? 

2. What factors may be limiting the use of FAR 13.5 Test Program for 
Certain Commercial Items procedures? 

C. ORGANIZATION 

Chapter I provides a brief discussion on the background of the FAR 13.5 Test 
Program for Certain Commercial Items, lists the research objectives, and explains the 
benefits of this report. 

Chapter II provides a literature review of the pertinent acquisition regulations and 
research conducted to date. The first section provides a history of FAR Subpart 13.5 
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through an examination of acquisition reform initiatives, Federal Acquisition 
Streamlining Act of 1994 (FASA) and the Federal Acquisition Reform Act of 1996 
(FARA) as well as the FAR Subpart 13.5 policies and procedures. The second section of 
this chapter discusses the importance and benefits of commercial item and simplified 
acquisition procedures and examines previous studies on FAR 13.5 usage rate. The last 
section provides a chronological review of the development of the Federal Procurement 
Data System-Next Generation (FPDS-NG) and the reports regarding its accuracy. 

Chapter III explains the methodology, scope, and limitations in a step-by-step 
manner that is designed to allow replication and improvements using additional data sets. 

Chapter IV presents the FPDS-NG data and provides analysis of the FAR 13.5 
test program usage rate, dollar value distribution, competition levels, number of offerors, 
and possible factors contributing to use or non-use of the FAR 13.5 test program. 

Chapter V summarizes the report’s findings by answering the research questions 
and provides recommendations for improving the usage rate of the test program. It also 
provides recommendations for future research studies. 

D. BENEFITS OF THE REPORT 

This report benefits federal agencies, contracting offices, contractors, small 
business. Congress, and ultimately U.S. taxpayers. This report explains the benefits of 
reduced administrative burden and cost savings associated with using the commercial 
item test program. The reduced workload directly benefits contracting offices and 
contractors, allowing them to reduce manpower and focus efforts on more valuable 
functions. This manpower and workload reduction translates into cost savings for 
companies and federal agencies, which can be passed on to the U.S. taxpayer. An 
accurate usage rate provides Congress with a sound data point necessary to make the 
FAR 13.5 test program permanent. They can save time and effort associated with the 
regularly renewal of the program and provide a stable tool for the contracting offices to 
use. Finally, additional acquisitions using simplified acquisition procedures opens up 
more opportunities for small businesses that are not capable of or willing to be involved 
in a large and complicated source selection process. 
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E. CHAPTER SUMMARY 


This chapter discussed the background of the FAR Subpart 13.5 Test Program for 
Certain Commercial Items by explaining the acquisition reform legislation leading to the 
program. It also described the research objectives by presenting two usage rate questions. 
The organization of the paper was then presented by explaining each of the contents of 
each chapter. Lastly, the benefits of this paper to federal agencies, contracting offices, 
contractors, small business. Congress, and taxpayers were provided. 
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II. LITERATURE REVIEW 


A. CHAPTER INTRODUCTION 

The purpose of this chapter is to provide a review of the literature directly related 
to the FAR Subpart 13.5 Test Program for Certain Commercial Items and the accuracy of 
FPDS-NG. A review of the program’s history will discuss acquisition reform legislation 
and the resulting FAR 13.5 policies and procedures. This discussion will be followed by 
an explanation of four conceptual program benefits and a review of previous studies 
aimed at determining benefits. Next, a chronological review of usage rate studies since 
2001 will be reviewed to examine various methodologies and resulting usage rates. 
Lastly, the history of events leading to the current state of FPDS-NG will be provided as 
well as past studies to discuss FPDS’ accuracy and reliability improvements. 

B. HISTORY OF FAR SUBPART 13.5, THE TEST PROGRAM FOR 
CERTAIN COMMERCIAL ITEMS 

1. Acquisition Reform: The Federal Acquisition Streamlining Act of 
1994 and Federal Acquisition Reform Act of 1996 

In response to years of unsuccessful attempts to simplify the acquisition process 
and reduce the excessive use of unique government specifications, the National Defense 
Authorization Act (NDAA) for Fiscal Year (FY) 1991 directed the Department of 
Defense (DOD) to establish what would become known as the Section 800 Panel (GAO, 
1993). This panel was tasked to 

(1) review the acquisition laws applicable to DOD with a view toward 
streamlining the defense acquisition process; (2) make any 
recommendations for the repeal or amendment of such laws as the Panel 
considered necessary; and (3) prepare a proposed code of relevant 
acquisition laws. (GAO, 1993, p. 3) 

Over the span of two years, the Section 800 Panel reviewed over 600 laws, 
ultimately producing a 1,800-page report containing more than 300 recommendations 
focused on “formulating changes to streamline and simplify the defense acquisition 
process and to improve the DOD’s capability to purchase commercial items and 
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technologies” (GAO, 1993, p. 3). Many of those recommendations served as the impetus 
for legislative reforms in the years to follow. 

The Federal Acquisition Streamlining Act of 1994 (FASA) and the Federal 
Acquisition Reform Act of 1996 (FARA) were two pieces of legislation that leveraged 
the recommendations of the Section 800 Panel to establish a foundation for increasing the 
acquisition of commercial items to meet the federal government’s needs. Signed into law 
a decade after the passage of the Competition in Contracting Act (CICA), which reshaped 
the competitive landscape of the acquisition community, the FASA’s and FARA’s 
reforms centered around reducing requirements and procedures within the acquisition 
process that were considered too costly, complex, and burdensome. 

The comprehensiveness of the FASA is best demonstrated by its sheer magnitude 
and breadth. The legislation totaled 167 pages and directed changes spanning across 
almost every conceivable acquisition area, specifically Contract Formation (Title I), 
Contract Administration (Title II), Service Specific and Major Systems Statutes (Title 
III), the Simplified Acquisition Threshold (SAT; Title IV), Acquisition Management 
(Title V), Other Procurement-Related Matters (Title VI), Small Business and 
Socioeconomic Laws (Title VII), Commercial Items (Title VIII), and even establishing a 
Federal Acquisition Computer Network (Title IX; FASA, 1994). In particular, the 
requirements of Title IV, Simplified Acquisition Threshold, and Title VIII, Commercial 
Items, provided significant reductions to the complexity of the federal acquisition system 
while also increasing its overall responsiveness. 

The FASA’s Title IV reforms to the SAT streamlined government acquisitions 

significantly. Although there were a number of legislative requirements under Title IV, 

the most notable change was the large increase to the SAT, which raised the ceiling from 

$25,000 to $100,000. In 1993, the DOD reported that “the lead time for awards less than 

$25,000 [was] 26 days [while] the lead time for awards above $25,000 [was] 90 days for 

simple sealed competitive bids and 210 days for negotiated competitive bids” (DOD, 

1993, p. 4). Furthermore, the report noted that a $100,000 threshold would allow the 

DOD to accomplish an estimated 99% of all contract actions using simplified acquisition 

procedures (DOD, 1993). Naturally, through dramatically expanding the number of 
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acquisitions under the SAT, the FASA’s reforms would drastically improve the speed and 
efficiency of the government’s acquisition process. 

Regarding increasing the acquisition of commercial items under Title VIII, the 
FASA changed the manner in which the government conducted routine business through 
numerous measures that aligned federal procurements with industry practices. First, the 
FASA established comprehensive definitions for the terms commercial item and 
nondevelopmental item to govern the applicability of the new statutes. Then, the FASA 
sought to reduce the cost and complexity for companies doing business with the 
government by limiting the clauses applicable to any contracts for commercial items and 
by exempting such acquisitions from other miscellaneous statutory requirements. Most 
importantly, however, the FASA established the “Preference for Acquisition of 
Commercial Items” under Section 8104, which required the government to proactively 
source commercial items by mandating, “to the maximum extent practicable, 
[requirements be defined] so that commercial items, or to the extent that commercial 
items suitable to meet the agency’s needs are not available, nondevelopmental items 
other than commercial items, may be procured to fulfill such requirements” (FASA, 
1994, § 2377(a)) and that procurement officials “acquire commercial items or 
nondevelopmental items other than commercial items to meet the needs of the agency” 
(FASA, 1994, § 2377(b)). By altering the way agencies define requirements, requiring 
market research, and mandating procurement officials seek commercial goods and 
service solutions first, the FASA moved federal acquisitions much closer to a market- 
based system. 

Two years later, the FARA was enacted into law as part of the NDAA for FY 
1996 under Division D to serve as a logical continuation of the FASA’s reforms, focusing 
particular attention on competition requirements and commercial item acquisitions 
(FARA, 1996). While the FARA contained a number of notable reforms to the federal 
acquisition system, one particular reform sought to expand on the success of both the 
increased SAT and the new preferences for commercial items. Section 4202, titled 
Application of Simplified Procedures to Certain Commercial Items, amended Section 
2304(g) of Title 10, United States Code, to establish the authority for the Test Program 
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for Certain Commercial Items. At the time, this program temporarily authorized the use 
of the highly successful Simplified Acquisition Procedures on requirements for 
commercial supplies and services up to $5 million. The law was incorporated into the 
FAR under Subpart 13.5 and titled Test Program for Certain Commercial Items. 
Established as a temporary program, the test program has generally been renewed every 
two years with some lapses in coverage. Most recently, in 2012, Congress allowed the 
test program to expire for a brief period before extending it again a year later until 
January 1, 2015, as part of the NDAA for FY 2013 (Defense Procurement and 
Acquisition Policy [DPAP], 2012). 

2. FAR Subpart 13.5 Policies and Procedures 

When active, the Test Program for Certain Commercial Items under FAR Subpart 
13.5, permits the 

use of simplified acquisition procedures for the acquisition of supplies and 
services in amounts greater than the simplified acquisition threshold but 
not exceeding $6.5 million ... including options, if the contracting officer 
reasonably expects, based on the nature of the supplies or services sought, 
and on market research, that offers will include only commercial items. 

(FAR 13.500(a)) 

In order to fully comprehend the potential impacts of the test program, this paper 
reviews what Simplified Acquisition Procedures are and how they differ from other 
acquisition methods. 

FAR Part 13 prescribes the policies and guidelines governing the use of 
Simplified Acquisition Procedures. More specifically, FAR Subpart 13.1, Procedures, 
and FAR Subpart 13.3, Simplified Acquisition Methods, contain a number of unique 
methods for contracting actions utilizing Simplified Acquisition Procedures. Those 
methods include the ability to 

• solicit quotes as well as offers (FAR 13.106), 

• use standing price quotations (FAR 13.103), 

• establish and order from blanket purchase agreements (FAR 13.303), 


8 



• “use any appropriate combination of the procedures in Parts 13, 14, 15, 35, 
or 36” (FAR 13.003(g)), and 

• “use innovative approaches, to the maximum extent practicable, in 
awarding contracts using simplified acquisition procedures” (e.g., reverse 
auction procedures; FAR 13.003(h)(4)). 

Regarding the contract file documentation for acquisitions conducted under the 
test program, the FAR states, 

the contract file must include (1) a brief written description of the 
procedures used in awarding the contract, including the fact that the test 
procedures in FAR Subpart 13.5 were used; (2) the number of offers 
received; (3) an explanation, tailored to the size and complexity of the 
acquisition, of the basis for the contract award decision; and (4) any 
justification approved under paragraph (a) of this section. (FAR 13.501(b), 

2014) 

Finally, the FAR requires that “for the period of this test, contracting activities 
must employ the simplified procedures authorized by the test to the maximum extent 
practicable” (FAR 13.500(b), 2014). The use of the term “maximum extent practicable” 
versus the clear mandate that would have been imposed by using the term “shall” 
indicates that contracting activities have a degree of flexibility when it comes to the use 
of the test program. Some circumstances that might reasonably justify foregoing the use 
of the Simplified Acquisition Procedures authorized by the program can be found under 
FAR 13.003(a), which include 

• sources of supply under FAR Part 8, 

• existing indefinite delivery/indefinite quantity contracts, or 

• other established contracts. 

Such exceptions to the test program make sense because they fall into either the 
category of mandatory use (e.g.. Federal Prison Industries [FPI] under FAR Subpart 8.6, 
Nonprofit Agencies Employing People Who Are Blind or Severely Disabled under FAR 
Subpart 8.7, Requirements Contracts under FAR 16.503) or a method that already 
provides a comparatively streamlined acquisition process (e.g.. Federal Supply Schedules 
under FAR Subpart 8.4 and Indefinite-Delivery Contracts under FAR Subpart 16.5); 
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however, acquisitions that seemingly forego the use of the Test Program for Certain 
Commercial Items without fulfilling the aforementioned criteria likely subject themselves 
needlessly to a more risky, complex process with larger procedural limitations. 

C. IMPORTANCE/BENEFITS OF FAR SUBPART 13.5 

The potential advantages of FAR Subpart 13.5 can be traced back to the general 
scope of FAR Part 13, which states. 

The purpose of this part is to prescribe simplified acquisition procedures 
in order to (a) Reduce administrative costs; (b) Improve opportunities for 
small, small disadvantaged, women-owned, HUBZone, and service- 
disabled veteran-owned small business concerns to obtain a fair proportion 
of Government contracts; (c) Promote efficiency and economy in 
contracting; and (d) Avoid unnecessary burdens for agencies and 
contractors. (FAR 13.002, 2014) 

It stands to reason then that the objective for the Test Program for Certain 
Commercial Items is to accomplish those same objectives, but for commercial item 
acquisitions in excess of $150,000 up to $6.5 million. Unfortunately, as the GAO (2014) 
found, “data are not specifically collected to assess test program benefits on the basis of 
such metrics as contracting lead time” (p. 14). Therefore, in order to assess the 
importance of the test program, this paper discusses the benefits that FAR Subpart 13.5 
provides from a conceptual standpoint and then reviews any findings from previous 
studies. 


1. Conceptual 

If utilized correctly, the test program should theoretically allow agencies to 
achieve the four previously identified benefits of simplified acquisition procedures 
(reduced administrative costs, improved small business opportunities, greater 
efficiencies, and reduced burdens), but on a broader scale. This section reviews each of 
those objective areas and how the procedures authorized by FAR Subpart 13.5 can 
uniquely contribute to each one. 
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a. Benefit #1-Reduced Administrative Costs 

First, the test program reduces administrative costs by minimizing documentation 
and prioritizing the efficiency of the acquisition process. In regard to reducing costs by 
limiting contract file documentation, FAR 13.106-3(b) instructs acquisition officials to 
“[k]eep documentation to a minimum. Purchasing offices shall retain data supporting 
purchases (paper or electronic) to the minimum extent and duration necessary for 
management review purposes (see subpart 4.8).” As previously mentioned, the specific 
documentation requirements for the test program are merely limited to four components 
consisting of 

a brief description of the procedures used in awarding the contract, ... the 
number of offers received, ... an explanation, tailored to the size and 
complexity of the acquisition, of the basis for the contract award decision; 
and [if necessary] ... any justification [for sole source or brand name 
acquisitions]. (FAR 13.501, 2014) 

Another way the test program minimizes administrative costs is through 
encouraging efficiency throughout the acquisition process. During the solicitation phase, 
simplified acquisition procedures instruct agencies to “respond to inquiries received 
through any medium (including electronic commerce) if doing so would not interfere 
with the efficient conduct of the acquisition” (FAR 13.106-1(f), 2014). Simplified 
acquisition procedures provide further efficiencies during the evaluation phase by 
requiring contracting officers to “ensure that quotations or offers can be evaluated in an 
efficient and minimally burdensome fashion” (FAR 13.106-2(b)(3), 2014). By 
encouraging (even requiring) efficiency when utilizing simplified acquisition procedures 
under the test program, contracting officers have a unique basis for supporting the 
reasonableness of decisions to eliminate any inefficiencies in both phases of the 
acquisition process, thereby reducing administrative costs. 

b. Benefit #2-Improved Small Business Opportunities 

Unlike acquisitions under the SAT, acquisitions conducted under the authority of 
the Test Program are not reserved exclusively for small business concerns (FAR 
13.003(b)(1)). Still, the use of simplified acquisition procedures seeks to increase the 
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opportunities for small businesses to participate in government contracts through 
reducing the complexity of the contracting process. Solicitations developed in accordance 
with FAR Part 13 procedures are generally shorter, simpler, and more understandable 
than FAR Part 14 or 15 solicitations. Such differences ultimately make government 
contracting opportunities more accessible, particularly for small businesses that may lack 
the staff and/or experience necessary to navigate through a more complex, 
comprehensive process. Furthermore, utilizing the test program allows any interested 
commercial small businesses to respond to solicitations, whereas Indefinite Delivery— 
Indefinite Quantity (IDIQ) contracts or General Service Administration (GSA) schedules 
limit opportunities only to those companies who have a previously established contract 
with the respective organization. Although such limitations are fair and in accordance 
with CICA, they still limit the potential pool of suppliers when compared to the complete 
openness of a combined synopsis-solicitation under the Test Program for Certain 
Commercial Items. 

c. Benefit #3-Greater Efficiencies 

In terms of promoting efficiency and economy in contracting, the test program 
accomplishes these goals in two main ways. First, the evaluation procedures prescribed 
by FAR Part 13 allow contracting officers the ability to significantly strea ml ine a process 
that would otherwise be exhaustive and time consuming for acquisitions in excess of the 
SAT. Specifically, FAR 13.106-2(b)(4) states. 

For acquisitions conducted using a method that permits electronic 
response to the solicitation, the contracting officer may—(i) After 
preliminary consideration of all quotations or offers, identify from all 
quotations or offers received one that is suitable to the user, such as the 
lowest priced brand name product, and quickly screen all lower priced 
quotations or offers based on readily discemable value indicators, such as 
past performance, warranty conditions, and maintenance availability; or 
(ii) Where an evaluation is based only on price and past performance, 
make an award based on whether the lowest priced of the quotations or 
offers having the highest past performance rating possible represents the 
best value when compared to any lower priced quotation or offer. 
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Comparing these processes to the comprehensive and restrictive procedures for 
opening and evaluating bids under FAR Subpart 14.4 or conducting source selections 
under FAR Subpart 15.3 easily illustrates the efficiencies afforded by the test program. 
For example, when utilizing the test program, contracting officers “are not required to 
establish a formal evaluation plan or competitive range, conduct discussions with 
vendors, or score quotations from offerors” (GAO, 2001, p. 3), all of which are more 
labor intensive than the procedures authorized by FAR Part 13. 

A second area where FAR Subpart 13.5 promotes economies throughout the 
procurement process is the ability to use innovative approaches when conducting 
acquisitions under the authority of the test program. More specifically, 

FAR 13.003(g) gives wide latitude in the procedures that can be used for 
simplified acquisitions by allowing “any appropriate combination” of the 
procedures in other parts of the FAR and encouraging the use of 
“innovative approaches, to the maximum extent practicable.” This had 
been held to permit the use of reverse auction procedures. (Cibinic, Nash, 

& Yukins, 2011, p. 1040) 

As stated within the guiding principles for the Federal Acquisition System, as 
long as members of the acquisition team exercise 

sound business judgment ... if a specific strategy, practice, policy or 
procedure is in the best interests of the Government and is not addressed 
in the FAR, nor prohibited by law (statute or case law). Executive order or 
other regulation...the strategy, practice, policy or procedure is a 
permissible exercise of authority. (FAR 1.102(d), 2014) 

Because simplified acquisition procedures naturally have fewer restrictions, they 
uniquely allow acquisition officials greater latitude to conduct acquisitions in a manner 
that is most economical for the government. 

d. Benefit #4-Reduced Burdens 

Finally, the test program avoids unnecessary burdens for agencies and contractors 
alike through the ability to issue requests for quotes (RFQs) instead of invitations for bids 
(IFBs) or requests for proposals (RFPs), which require interested parties to submit a 
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binding offer in the form of a bid or proposal, respectively. The RFQ process 
differentiates itself from IFBs and RFPs because 

a quotation is not an offer and, consequently, cannot be accepted by the 
Government to form a binding contract ... [instead], the order is an offer 
by the Government to the supplier ... [and] a contract is established when 
the supplier accepts the offer. (FAR 13.004) 

Thus, RFQs have the unique benefit of providing “a substantially lower risk for 
the supplier than that which would be incurred if the supplier were making an offer” 
(Cibinic et ah, 2011, p. 1036). Similarly, “an RFQ also gives the agency more flexibility 
in conducting a procurement” (Cibinic et ah, 2011, p. 1036) because the government can, 
for example, use standing price quotations as the basis for a contract, which further 
reduces the burden on the government. Furthermore, the government is not bound by 
RFQs to either award a purchase order or cancel the solicitation, which further reduces 
burdens on government agencies (FAR 15.402(e), 2014). Still, if the situation warrants 
the use of an RFP or an IFB to bind a contractor through the submission of an offer, the 
test program allows the government to use those approaches as well. Table 1 summarizes 
the benefits previously addressed in this section. 


Table 1. Summary of FAR Subpart 13.5 Benefits 


1. Reduced Administrative Costs 

Limiting contract file documentation 

More efficient acquisition process 

2. Improved Small Business Opportunities 

Shorter, simpler solicitations under SAP are likely more easily understood 

Allows any eligible, interested company to respond vs. IDVs, which are exclusive 

3. Greater Efficiencies 

Permits a streamlined evaluation process 

Allows the use of innovative approaches, such as reverse auctions 

4. Reduced Burdens 

Allows use of RFQs, which are generally less burdensome 

Still permits the use of RFPs and IFBs, if warranted 
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Ultimately, the various policies and procedures governing the Test Program for 
Certain Commercial Items allows acquisition officials unique flexibilities to be more 
efficient and open with procurements. To what extent, specifically, is a question that 
previous studies have attempted to answer. 

2. Previous Studies 

Researchers have attempted to quantify the benefits of the commercial item test 
program since at least 1999, when “the Office of Federal Procurement Policy (OFPP) 
surveyed procurement executives in federal agencies to obtain their opinions on the 
benefits” (GAO, 2001, p. 4). Although early GAO reports struggled to find empirical data 
to determine the benefits of the program, recent studies have shown the program’s worth. 
The following is a summary of the studies that have been accomplished to date. 

One of the earliest measures of the commercial test program’s benefits came in 
the form of a survey conducted in 1999 by the OFPP (GAO, 2001). The OFPP polled 
government procurement executives and received responses that claimed several believed 
benefits. The benefits included “a positive impact on (1) time required to award a 
contract, (2) administrative costs, (3) prices, (4) small business participation, and (5) 
delivery of products and services” (GAO, 2001, p. 4). Although this initial report on the 
program was positive, the GAO (2001) quickly pointed out some key weaknesses about 
the survey. Some weaknesses included that 

the survey did not measure the extent to which (1) time required to award 
contracts was reduced, (2) administrative costs were reduced, (3) prices 
reflected the best value, (4) small business participation was promoted, or 
(5) delivery of products and services was improved. (GAO, 2001, p. 4) 

The GAO’s ultimate conclusion in the 2001 report was that despite the federal 
procurement executives’ desire to make the program permanent, benefits of the program 
have not yet been clearly demonstrated. The GAO recommended that Congress require 
that the OFPP construct a new methodology of testing the program’s benefits. Congress 
responded by mandating that the GAO determine the extent to which the DOD was using 
the program and any resulting benefits (GAO, 2003). The GAO issued a report in 2003 
that stated although interviews with procurement officials showed favorable program 
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results, the GAO was unable to accomplish either mandated task due to a lack of reliable 
data (GAO, 2003a). The GAO criticized the reliability of FPDS data and pointed out 
federal agencies’ lack of progress in gathering useable data on the program. 

In 2006, a significant research study was conducted on the Navy Fleet and 
Industrial Supply Center’s (FISC) use of the test program (Johnson, Simonson, & 
Ziegler, 2006). The results quantified benefits using transaction touch time and ultimately 
determined a 90% reduction in time and cost when using the program (Johnson et al., 
2006). The study found “an average cost reduction of over $9,500 per transaction” 
(Johnson et al., 2006, p. 21) when using FAR 13.5 instead of traditional protocol. The 
study was one of the first to actually quantify the benefits of the test program. 

In 2008, the DOD provided a report to Congress outlining several benefits of the 
test program. These benefits included “increased competition and small business 
participation, decreased acquisition lead time, enhanced workforce morale, and decreased 
costs all while increasing the war fighter’s capabilities” (DOD, 2008, p. 7). The DOD’s 
(2008) report went to on to express the department’s desire to make the program 
permanent, stating, “Incremental extension of the Test Program is a barrier to increasing 
the use of commercial items” (p. 7). After the test program’s brief expiration in 2012, the 
DOD (2012) provided another report that stated, “Expiration of the FAR 13.5 Test 
Program for Certain Commercial Items resulted in an additional barrier to competition for 
actions greater than the simplified acquisition threshold” (p. 13). 

In 2014, the GAO developed a methodology to assess the test program and 
formally acknowledged its benefits in a report to Congress. The GAO’s (2014) 
performance audit “selected a nonrandom, nongeneralizable sample” (p. 2) from the 
DOD, DOI, and DHS, and determined that the test program “reduced contracting lead 
time and administrative burdens and generally did not incur additional risks above those 
on other federal acquisition efforts for those contracts” (GAO, 2014, para. 2). This report 
was significant because it represented the first time the GAO offered a quantitative 
response to Congress’ mandate regarding the test program’s benefits. 
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As discussed in previous paragraphs, past GAO reports regularly stated that 
reliable data was an issue in determining the program’s benefits. As explained in Section 

C, the same challenge existed in the mandate to determine an accurate usage rate. 

D. PREVIOUS USAGE RATE STUDIES 

As early as 2001, researchers, such as the GAO, have attempted to quantify the 
use of the FAR Subpart 13.5, Test Program for Certain Commercial Items, and measure 
the program’s use. Studies on several agencies have produced varied results. This 
variation is largely due to each study using its own methodology and data set. The 
following section provides a chronological literature review of the studies’ methodologies 
and results. The chronological order traces the change in research design and usage rates 
throughout the last 15 years. 

The NDAA for FY 2000 first mandated that the GAO evaluate the FAR Subpart 
13.5, Test Program for Certain Commercial Items (GAO, 2001). The GAO approached 
the mandate by reviewing 12 total contracts that used the test program at the Air Force’s 
Air Armament Command at Eglin Air Force Base, the Army’s Defense Supply Service- 
Washington, and the Navy’s Fleet Industrial Supply Center-NorfoUc. The GAO’s (2001) 
research design did not attempt to identify a usage rate, and “data was not collected to 
provide a basis for measuring whether the test program produced the desired results” (p. 
4). Although the GAO referenced an Office of Federal Procurement Policy (OFPP) 
survey that showed procurement executives’ belief in the effectiveness of the program, 
empirical data was not collected, and the GAO (2001) ultimately concluded that the 
benefits could not yet be clearly demonstrated. 

A 2005 Naval Postgraduate School (NPS) joint applied project examined the 
Marine Corps’ Regional Contracting Office-Southwest (RCOSW), located at Camp 
Pendleton, to determine whether the office was fully utilizing simplified acquisition 
legislation, such as those offered in FAR Subpart 13.5 (Gillespie, 2005). The project did 
not use Federal Procurement Data System (FPDS) data. Instead, it used data pulled from 
RCOSW’s awards database for FY 2003, 2004, and 2005. The study did not provide a 
FAR Subpart 13.5 usage rate but showed that while 90% of RCOSW’s spending was 
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eligible for simplified acquisition procedures (SAP), such as the test program, only 66% 
used SAP. The study concluded that the remaining actions were procured “using 
traditional contract methods, even though they fit the criteria of the FAR 13.5 test 
program and could have been bought using SAP” (Gillespie, 2005, p. 25). The benefits of 
this study are that it utilized a more data-driven and quantifiable research approach than 
the 2001 GAO report and avoided any FPDS accuracy issues by using the RCOSW’s 
local system. 

In 2006, an Acquisition Research Program (ARP) report by E. Cory Yoder used 
FPDS data to empirically analyze the FAR 13.5 test program usage rate in the Navy. The 
study examined the Naval Supply Systems Command (NAVSUP) and the Navy’s Fleet 
and Industrial Supply Center-San Diego (FISCSD) through both FPDS and contract file 
review to determine the extent of FAR 13.5 program use. The analysis identified usage 
rates from FPDS (as shown in Table 2) and utilization effectiveness through contract 
inspection. The usage rates by dollar value varied significantly from 13% to 56% while 
the usage rates by action were fairly consistent between NAVSUP and FISCSD and were 
as high as 57% in FY 2001 (Yoder, 2006). Despite the rather positive usage rates, 
contract inspection at FISCSD showed that contracts coded as using the FAR 13.5 test 
program often did not fully utilize all advantages provided by the program (Yoder, 2006). 
This study’s methodology effectively utilized FPDS to derive reliable usage rates, but 
also recognized the limitations of FPDS leading up to 2006 and utilized contract 
inspection to provide a more complete analysis. 
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Table 2. NAVSUP and FISCSD FAR 13.5 Utilization Rate 

(after Yoder, 2006) 


ORGANIZATION 

DATA SET YEAR 

USAGE RATE ($) 

USAGE RATE 
(ACTIONS) 

NAVSUP 

EY 2001 

36% 

45% 

NAVSUP 

EY 2002 

23% 

28% 

NAVSUP 

EY 2003 

24% 

28% 

NAVSUP 

EY 2004 

34% 

46% 

NAVSUP 

EY 2005 

27% 

28% 

Navy FISCSD 

EY 2001 

13% 

57% 

Navy FISCSD 

EY 2002 

15% 

20% 

Navy FISCSD 

EY 2003 

37% 

38% 

Navy FISCSD 

EY 2004 

56% 

53% 

Navy FISCSD 

EY 2005 

26% 

19% 


In 2011, a Defense Acquisition University (DAU) study leveraged Yoder’s work 
to examine FAR 13.5 test program utilization at the Defense Logistics Agency (DLA; 
Shearer, 2011). This research took advantage of the improvements to the Federal 
Procurement Data System-Next Generation (FPDS-NG) data accuracy and the newly 
added FPDS-NG data element lOJ to derive a usage rate (by action) of 6% for FY 2009 
and 11% for FY 2010 (Shearer, 2011). However, the data mining methodology did not 
appear to have controlled for actions that were ineligible for the program, such as limits 
by dollar thresholds. Therefore, the usage rates provided in Table 3 appear to answer the 
basic question of how often the FAR 13.5 test program is used in DLA, regardless of 
whether the action was eligible. 
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Table 3. Defense Logistics Agency FAR 13.5 Utilization Rate 

(after Shearer, 2011) 


AGENCY 

DATA SET YEAR 

USAGE RATE ($) 

USAGE RATE 
(ACTIONS) 

DEA 

EY 2009 

not provided 

6% 

DEA 

EY 2010 

not provided 

11% 


While the DAU was leveraging Yoder’s 2006 study, Yoder and Hawkins (2011) 
were developing a new methodology to derive the FAR 13.5 utilization rate using FPDS- 
NG data for the Air Force and Navy. Instead of relying strictly on the new FPDS-NG 
Data Element lOJ, their research used the ninth digit of the solicitation number as an 
alternative indicator (Yoder & Hawkins, 2011). After controlling for factors that would 
make an action ineligible for the test program, such as dollar thresholds, modifications, 
delivery orders, and non-commerciality, their research (depicted in Table 4) showed a 
78% and 79% usage rate for the Air Force and Navy, respectively (Yoder & Hawkins, 
2011 ). 


Table 4. Air Force and Navy FAR 13.5 Utilization Rate 
(after Yoder & Hawkins, 2011) 


AGENCY 

DATA SET YEAR 

USAGE RATE ($) 

USAGE RATE 
(ACTIONS) 

Air Eorce 

EY 2010 

not provided 

78% 

Navy 

EY 2010 

not provided 

79% 


A comparison between Yoder and Hawkins’s research and a recent study by the 
GAO in 2014 highlights the strong effect that methodology design can have on the FAR 
13.5 test program usage rate. The GAO examined FY 2011 FPDS-NG data from the 
DOD, DOI, and Homeland Security and determined the usage rate was 9% by dollar 
value and 12% by total actions (GAO, 2014). Additionally, the report determined the 
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usage rate for the entire federal government was only 2%. Complete summaries of the 
GAO’s findings are consolidated in Table 5. The GAO report methodology relied 
heavily on the FPDS-NG Data Element lOJ and, unlike Yoder and Hawkins, was 
controlled solely by dollar threshold to remove ineligible or impractical actions. 


Table 5. Total Federal, DOD, DOI, and DHS FAR 13.5 Utilization Rate 

(after GAO, 2014) 


AGENCY 

DATA SET 
YEAR 

USAGE RATE 
($) 

USAGE RATE 
(ACTIONS) 

RESEARCH 

GROUP 

Federal 

Government 

FY 2011 

2% 

Not Provided 

GAO 

DOD, DOI, & 
DHS 

FY 2011 

9% 

12% 

GAO 

Army Materiel 
Command 

FY 2011 

6% 

7% 

GAO 

Naval Supply 
Systems 
Command 

FY 2011 

23% 

26% 

GAO 

Air Force 
Materiel 
Command 

FY 2011 

6% 

6% 

GAO 

U.S. Coast 
Guard 

FY 2011 

17% 

24% 

GAO 

DOI 

Acquisition 

Services 

Directorate 

FY 2011 

18% 

22% 

GAO 


The large variation in the usage rate shown by the last 15 years of research is due 
largely to the variation in methodology design for each study. The change in 
methodology over the years coincides with the change in availability and accuracy of the 
data set, FPDS-NG. As FPDS-NG improved in capability and accuracy, usage rate 
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studies applied different approaches to answer the FAR 13.5 usage rate question. The 
next section discusses the development of FPDS-NG, the improvements over the years, 
and the studies conducted regarding FPDS-NG accuracy. 

E. USE OF THE FEDERAL PROCUREMENT DATA SYSTEM-NEXT 

GENERATION 

The FPDS-NG system has made several significant improvements over the 
years. Those improvements have enhanced the ability of researchers to conduct studies 
quantifying the test program’s usage and overall benefits. In conducting those studies, 
researchers used various methodology designs and data sets, which resulted in varying 
usage rates. This section explains the history of events leading to the current state of 
FPDS-NG as well as past studies to discuss FPDS’ accuracy and reliability 
improvements. 

1. Chronological Development of FPDS-NG 

The FPDS-NG is an automated web-based information system used to develop, 
collect, and disseminate federal procurement data (FPDS-NG, n.d.). It is the only 
approved federal procurement data repository authorized for use by Congress, the 
Executive Branch, audit agencies such as the GAO, private sector companies, and the 
general public (FPDS-NG, 2014). FPDS was established under the direction of the 
OFPP, as amended, 41 U.S.C. 401 et seq. and the FAR Subpart 4.6 (FPDS-NG, 2014) 
and has served as the federal government’s primary depository for contract data since 
1978 (GAO, 2003b). This report uses FPDS-NG as its primary data source for evaluating 
the usage rate of FAR Subpart 13.5, Test Program for Certain Commercial Items. 

The following time line provides a brief history of FPDS-NG and a summary of 
the events leading to the current version of FPDS-NG: 

• December 1972: The Commission on Government Procurement 
determined that the government should have a single system responsible 
for collecting and reporting on government procurement statistics (GAO, 
1980). 
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• November 1973: An interagency task group was charged with studying the 
commission’s recommendation and obtain agency comments (GAO, 
1980). 

• August 1974: “Congress passed the Office of Federal Procurement Policy 
Act (Public Law 93-400)” (GAO, 1980, p. 2) requiring the OFPP 
administrator to establish a procurement tracking system (GAO, 1980). 

• February 1978: The OFPP administrator designated the DOD as OFPP’s 
executive agent in charge of establishing and operating the Federal 
Procurement Data Center (FPDC) and FPDS (GAO, 1980). 

• October 1978: Federal agencies were required to start collecting identified 
data for FPDS (GAO, 1980). 

• October 1979: Congress enacted Public Law 96-83 transferring 
responsibility of the FPDC from the DOD to the General Services 
Administration (GSA; GAO, 1980). 

• April 2003: The GSA awarded a contract to Global Computer Enterprises 
(GCE), Inc., to review and transfer federal agencies’ data to EPDS-NG. 
The contract also required that GCE validate and connect agencies 
contract writing systems to EPDS-NG (GAO, 2005). 

• October 2003: EPDS-NG became operational (GAO, 2003b). 

• December 2004: EPDS-NG was available for public use (GAO, 2005). 

• September 2006: “The Eederal Eunding Accountability and Transparency 
Act of 2006 (EEATA, P.E. 109-282) mandated the development of a user- 
friendly system comprising a variety of government spending data, 
including procurement data” (Halchin, 2013, p. 3). The user-friendly 
searchable website developed was usaspending.gov. 

• March 2007: The 0MB issued a memorandum to start the Federal 
Procurement Data Verification and Validation (V&V). 

• December 2007: Usaspending.gov was officially launched (USA 
Spending, n.d.). 

• 2009: Data Eield lOJ, “Commercial Item Test Program” was added to 
EPDS-NG (GAO, 2014). 

The most significant upgrade from EPDS to EPDS-NG included more “machine- 
to-machine” automated data entry methods and error detection software designed to 
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provide immediate data verification (GAO, 2003b). Automated data entry allowed the 
agencies’ contract writing systems to directly transfer contract data to FPDS-NG, and the 
error detection software enabled the user to correct mistakes prior to the data being 
finalized. Both of these features were included in an effort to improve the reliability of 
the data (GAO, 2003b). 

2. FPDS-NG FAR Subpart 13.5 Coding Accuracy 

Reliability of FPDS data has been an issue since the program’s inception. The 
OFPP and GSA have continually focused on making improvements to the system over 
the years and have made great strides since agencies first started providing data in 1978. 
Currently, OFPP and GSA require each agency to complete an annual certification of 
accuracy and completeness percentages of their FPDS-NG data (FPDS-NG, 2014). This 
section discusses past FPDS-NG data accuracy issues and the improvements made to the 
system. 

Less than a year after agencies started reporting procurement data, the 
Subcommittee on Human Resources and the Committee on Post Office and Civil Service 
requested that the comptroller general examine the effectiveness of FPDS (GAO, 1979). 
Despite 56 agencies being required to report, 20 agencies had either “not reported, 
reported in part, or submitted reports not in accordance with prescribed instructions” 
GAO, 1979, p. 1). The comptroller general reported that 

the Federal Procurement Data System relies on the integrity of many 
individuals to prepare reports and to prepare them correctly. If, for some 
reason, a report is not prepared, the data on the contract award will not 
enter the system. Furthermore, the Center has no means for knowing 
whether data are reported for all contracts. (GAO, 1979, p. 2) 

The comptroller general and the GAO quickly followed this 1979 report with their 
own FPDS examination. This 1979 report was quickly followed by another FPDS 
examination by the comptroller general and the GAO. In April of 1980, they reported 
that, while progress was made setting up the system, “FPDS failed to alert users that the 
system contained known errors” (GAO, 2003b, p. 2). The report also stated that 
“timeliness of reporting by the agencies and accuracy of the data submitted” needed 
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improvement (GAO, 1980, p. ii). FPDS was in its infancy and was already facing tough 
scrutiny. 

Inspections and reports addressing FPDS accuracy and overall system 
improvement continued to be a regular occurrence for the OFPP and GSA as they worked 
to improve the system. A 1994 GAO report documented a meeting between the GAO, 
OFPP, and GSA regarding improvement recommendations. One of the report’s biggest 
improvement suggestions involved developing accuracy and completeness standards for 
FPDS data. The OFPP and GSA agreed with the GAO’s recommendations and tasked the 
Federal Procurement Data Center (FPDC) to establish a set of standards for distribution 
to all agencies (GAO, 1994). 

Despite improvements by the OFPP and GSA, accuracy issues continued to be a 
problem for the FPDS. For instance, in a study to examine the usefulness of the 
commercial item test program, the GAO (2001) examined procurement data and initially 
determined that the federal government “purchased about $31.6 billion in commercial 
items for fiscal year 2000 and used the simplified procedures to purchase $1.9 billion in 
commercial items” (p. 3). However, interviews with the actual agencies showed that the 
number of contracts using the test program was overstated. The GAO (2001) was not able 
to evaluate the full extent of the possible over-reporting problem. 

One of the most significant reports to highlight accuracy issues in FPDS was a 
September 2003 GAO report that attempted to examine the usage of the commercial item 
test program for the DOD, Department of Treasury (DOT), and Department of Justice 
(DOJ). The report found FPDS data to include significant errors and came to the overall 
assessment that “there is no reliable information for measuring the test program’s 
benefits” (GAO, 2003a, p. 7). Figure 1 is an excerpt from GAO report GAO-03-1068 and 
shows the substantial over and understatement of the value of the contracts that FPDS 
showed as using the commercial item test program. For example, FPDS showed that the 
DoT’s U.S. Mint used the program for $242 million in contract actions for FY 2001. 
However, through interviews with DoT procurement officials, the GAO (2003a) 
determined that U.S. Mint did not use the commercial item test program that year. 

Conversely, FPDS showed a $4 million value for the DLA, only to determine through 
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interviews with DLA procurement officials that the value was actually $146 mi llion 
(GAO, 2003a). 


Department's 

buying 

organization 

Value of test program 
contracts, according to 
FPDS (fiscal year 2001) 

Wbat procurerrtent officials 
said about FPDS's data 

Deparlmenl of the 
Treasury's U.S. Mint 

S242 million 

U.S. Mint said it did not use the 
test program at all 

DOO's Defense 

Logistics Agency 

S4 million 

Defense Logistics Agency said 
it obligated SI46 million in test 
program contracts 


S118 million After reviewing portons of 

FPDS's data, about S31 miiiion 
in contract actions. Federal 
Prison Industnes said none of 
those items were purchased 
under the test program 

Figure 1. Examples of FPDS Discrepancies (from GAO, 2003a, p. 5) 

Two months after the September report, the GAO sent a letter to the director of 
the 0MB summarizing examples of FPDS data reliability issues found in past GAO 
studies (GAO, 2003b). The letter recognized that a major cause of the errors in FPDS was 
data entry mistakes by the agencies’ contracting personnel. It also highlighted the 
positive attributes of the new FPDS-NG that addressed data entry errors and made the 
following three recommendations for the improvement of data reliability in the new 
system: 

1. Ensure that agencies allocate the resources necessary to implement 
contract writing systems capable of electronic transfer of information to 
FPDS-NG. 

2. Require agencies that have not yet implemented electronic contract 
writing systems to report regularly to the 0MB on their plans to ensure 
reliability of the information reported to FPDS-NG. 

3. Request that major agencies, in consultation with the GSA, conduct 
regular reviews of their procedures for collecting and reporting 
information to FPDS- NG. Agencies should conduct such reviews 
annually until a satisfactory level of reliability is achieved, and 
periodically thereafter. (GAO, 2003b, p. 3) 


Deparlmenl of 
Justice's Federal 
Prson Industries 


The 0MB and GSA both agreed with the recommendations, and on August 25, 
2004, the 0MB sent a memo to the heads of executive departments and agencies and the 
President’s Management Council directing them to take steps necessary to comply with 

26 




the GAO’s recommendations (Johnson, 2004). The upgrade to FPDS-NG and the 
required actions in this memo represented a significant improvement of procurement data 
reliability. 

In September 2005, the GAO issued another letter to the 0MB summarizing a 
review of the FPDS-NG improvements to date (GAO, 2005). The report raised concerns 
about federal agencies’ lack of progress toward accuracy and timeliness of data, which 
largely stemmed from contract writing systems not yet electronically interfacing with 
FPDS-NG. While the report noted that 90% of agencies made the connection, it 
highlighted the DOD specifically for its implementation delays (GAO, 2005). The GAO 
report recommended that the 0MB work directly with the DOD to connect its contract 
writing systems to FPDS-NG. A memorandum from the director of Defense Procurement 
and Acquisition Policy (DPAP) to the defense agencies dated May 1, 2007 cited the 
challenges with the transition to FPDS-NG, but asserted that “the majority of DOD 
contracting offices are now reporting their FY07 contract actions to FPDS-NG” (DPAP, 
2007, p. 1). While adjustments and improvements to FPDS-NG were ongoing, the 
connection of the DOD’s contract writing systems was a significant step in finalizing the 
transition to the new system and overall; 2007 showed increased focus and progress 
toward the goal of data accuracy. 

A significant example of increased focus and progress toward data accuracy is 
displayed in the 0MB’s (2007) Federal Procurement Data Verification and Validation 
(V&V) program that required agencies to “[ejstablish an agency-wide requirement for 
routine, statistically-valid data verification and validation” (Denett, 2007, p. 1) and 
provide an annual certification of their FPDS-NG data. The 0MB issued a memorandum 
on May 9, 2008, which provided guidance on improving data quality and gave further 
instructions on the new V&V program. In response to the 0MB’s V&V program and 
overall focus on FPDS-NG data quality, the DOD created an FPDS Contract Reporting 
Data Improvement Plan, which included a 10-step process for data V&V (DPAP, 2008). 
The plan was sent to the 0MB and all DOD agencies for implementation on July 25, 
2008 (DPAP, 2008). 
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As the transition to FPDS-NG continued and focus on data accuracy V&V 
increased, the studies of the Test Program for Certain Commercial Items persisted. In an 
effort to improve data collection on the use of the test program, the DOD requested an 
FPDS-NG change (GAO, 2014). “FPDS-NG was modified in 2009 to include a specific, 
single reporting field to identify test program contracts” (GAO, 2014, p. 9). The change 
resulted in the addition of data element lOJ, Commercial Item Test Program (FPDS-NG, 
2014). 

In 2009, the GAO (2009) recognized the improvements of FPDS-NG but 
continued to report data quality concerns with the system. A September 2009 GAO report 
noted that 

submitting data electronically has improved the reliability of FPDS-NG, 
and while we have found some FPDS-NG data sufficiently reliable for 
specific reports since our last review of the system in 2005, recent GAO 
reports illustrate that the quality of some FPDS-NG data remains a 
concern, (p. 4) 

This mixed review highlighted the progress made through electronically 
submitted contract data, referencing a GSA report that stated that “more than 99 percent 
of data in FPDS-NG data were being submitted to the system electronically and that the 
agencies submitting the data had reviewed and verified the accuracy and completeness of 
their data” (GAO, 2009, p. 4). While the overall report voiced data reliability concerns 
about several government systems, including FPDS-NG, the report included positive 
comments about the data accuracy improvements made with FPDS-NG. 

As data reliability improved, the 0MB continued to push for accurate, complete, 
and timely procurement data through its V&V program. In May 2011, the 0MB issued 
the memorandum Improving Federal Procurement Data Quality: Guidance for Annual 
Verification and Validation, which expanded on the reporting requirements of FAR 4.604 
and looked to standardize the V&V process by providing sampling methodologies and 
reporting templates (Gordon, 2011). 

FPDS-NG data reliability improved greatly since its inception in 1978. A USA 
Spending (n.d.) report titled Federal Government Procurement Data Quality Summary: 
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Fiscal Years 2008 through 2011—For Agency Data in the Federal Procurement Data 
System found that 98.3% of contract actions from 2008 through 2011 were timely and 
94% of the prescribed data elements were accurate. While room for improvement 
remained, clearly, FPDS-NG coding accuracy was headed in the right direction. This 
positive outlook is supported by a 2013 report by the Center for Strategic & International 
Studies (CSIS), which found that while issues remain to be addressed, “Continued efforts 
by both legislative and user-generated feedback to FPDS-NC have led to measurable 
changes in data quality” (Dadsetan & Raghavan, 2013, p. 2). It continued, “The 
combination of bottom-up and top-down efforts signals a promising shift in the quality 
and quantity of future government contracting data management, as better data attracts 
larger user-bases and generates more feedback” (Dadsetan & Raghavan, 2013, p. 2). The 
positive outlook presented in these statements shows significant progress made. 

Finally, the most significant, current, and pertinent report on FPDS-NC data 
reliability came from the CAO in February of 2014. The report specifically assessed the 
reliability of the commercial item test program data in FPDS-NC by randomly sampling 
243 contracts from FY 2011 and requiring the DOD, DOI, and Homeland Security to 
verify the FPDS-NC coding (GAO, 2014). The departments were then required to submit 
contract file documentation proving use of the commercial test program. From the 
sample, the GAO (2014) “determined that the test program data reported in FPDS-NG 
were sufficiently reliable to determine the minimum extent to which the test program was 
used by the components” (p. 27). This study did not, however, examine contracts that 
were not coded as using the commercial item test program and therefore “did not assess 
the extent to which test program actions and obligations may be underreported” (GAO, 
2014, p. 27). Therefore, this study shows that contracting officials are generally not 
erroneously coding purchases using the commercial item test program and overstating the 
use, but it does not address the possibility of understatement through the lack of coding 
actual test programs contracts. 

Although there are still opportunities to improve FPDS-NG, the latest reports 
show accuracy has increased substantially, particularly with FAR 13.5 test program data. 
In addition to FPDS-NG being the only approved federal procurement data repository 
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authorized for use by Congress since 1978 (GAO, 2003b), the recent improved accuracy 
make it the most logical and sound data source available for studying the FAR 13.5 usage 
rate in the Air Force. 

F. CHAPTER SUMMARY 

Previous literature pertaining to FAR Subpart 13.5 Test Program for Certain 
Commercial Items shows that a program meant to simplify the acquisition process has a 
complex history. The program’s usage and overall benefits have been questioned since its 
inception by the Federal Acquisition Streamlining Act of 1994 and Federal Acquisition 
Reform Act of 1996. While a review of FAR 13.5 policies and procedures showed the 
program has a clear goal of providing agencies four fairly straightforward benefits of 
simplified acquisition procedures (reduced administrative costs, improved small business 
opportunities, greater efficiencies, and reduced burdens), the complexity of a simple 
program arises when groups, such as GAO and academic researchers, attempt to quantify 
these benefits and determine overall usage rates. Specifically, early GAO reports 
struggled to find empirical data to determine the benefits of the program. This was 
largely due to reliability issues with the federal governments’ contract reporting tool, 
FPDS-NG. However, as FPDS-NG improved over the years, the ability to quantify the 
test program’s usage and overall benefits improved as well and recent studies have shown 
the program’s value. In conducting those studies, researchers used various methodology 
designs and data sets, which resulted in varying usage rates. Chapter III explains how this 
paper’s methodology builds on those past studies and leverages the improvements of 
FPDS-NG. 
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III. METHODOLOGY 


A. CHAPTER INTRODUCTION 

The purpose of this chapter is to explain the methodology and limitations 
associated with this research. First, this chapter will explain how the literature review was 
used to create a research schema devised to improve on past FAR 13.5 test program 
studies. A detailed step-by-step process, using the U.S. Air Force data from February 1, 
2013, to May 30, 2014, will be provided in a manner that allows duplication with other 
data sets. Lastly, three limitations associated with this research are identified and 
discussed. 

B. METHODOLOGY 

The research methodology for this project consists of a literature review and 
analysis of FPDS-NG data to determine the usage rate of the Test Program for Certain 
Commercial Items. The literature review examined legislature and federal government 
policy behind the establishment of the test program, the design and benefits of the 
program, previous usage rate studies, and accuracy of the FPDS-NG. The analysis of 
FPDS-NG data uses a schema designed to control for factors such as dollar thresholds 
and mandatory sources to capture the most accurate test program usage rate for the U.S. 
Air Force from February 1, 2013, to May 30, 2014. The precise step-by-step process and 
rationale is outlined in the following section. 

The data set contained all Air Force contract actions reported in FPDS-NG from 
February 1, 2013, to May 30, 2014. The data was located in an Excel file, and the 
following sorting and elimination steps were conducted: 

Step I: Sort Column FJ (Base and all Options Value—Data Element 3A) by dollar 
amount. 

Rationale: According to the FPDS-NG “User Manual” (2014), 

This data element is required for all Awards and Modifications. Enter the 

mutually agreed upon total contract or order value including all options (if 

any). Eor modifications, this is the change (positive or negative, if any) in 
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the mutually agreed upon total contract value. See Data Dictionary 
Element 3A Use Case for appropriate data entry requirements, (p. 22) 

FAR Subpart 13.5, Test Program for Certain Commercial Items, applies to 
contract actions for the “acquisition of supplies and services in amounts greater than the 
simplified acquisition threshold but not exceeding $6.5 million ($12 million for 
acquisitions as described in 13.500(e)), including options” (FAR Subpart 13.5, 2014). 
Sorting by data element 3A ensures that the data is sorted by the dollar amount that 
captures the base and all options instead of strictly the original obligation amount. 

Step 2: Eliminate all actions less than $150,000 and more than $6.5 million. 

Rationale: Per FAR Subpart 13.5, the Test Program for Certain Commercial Items 
is only applicable to procurements in this price range. Removing all actions above or 
below those thresholds ensures that actions that are ineligible for the program are 
excluded from the data set. 

Step 3: Filter Column G (Modification Number—Data Element IB) and remove all 
modifications. 

Rationale: According to the FPDS-NG “User Manual” (2014), 

Enter a Modification Number in accordance with your agency’s policy, 
when reporting modifications to contracts, agreements, or orders. This 
number must be unique for any given PHD. This is a required field for all 
modifications and can be up to 25 characters. Otherwise, leave blank, (p. 

19) 

FAR Subpart 13.5, Test Program for Certain Commercial Items, is not applicable 
to modifications. Removing all actions with input in this field eliminates ineligible 
modifications from the data set. 

Step 4: Filter Column I (Referenced IDV/PIID—Data Elements 1C, IG, and IH) 
and remove each action that has input in the cell. 

Rationale: According to the FPDS-NG “User Manual” (2014), 

This data element is required for all Delivery/Task orders and BPA calls. 

It is not required for a DCA, Purchase Order, and a Modification. Enter 
the contract or agreement number of the IDV against which your order is 
placed. FPDS-NG allows for up to 50 characters. This data element 
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consists of three parts: The Referenced IDV agency identifier, Referenced 
IDV PHD and the Referenced IDV Modification Number, (p. 19) 

This data element is only populated when the action is a delivery order, task 
order, or blanket purchase agreement (BPA) call against an existing contract or 
agreement. Writing a new contract using the Test Program for Certain Commercial Items 
when a contract is already in place to provide the supply or service is not an efficient or 
effective use of the program. Removing all actions that have input for this data element 
eliminates actions that already had a contract vehicle in place for use. 


Step 5: Filter Column AR (Commercial Item Acquisition Procedures—Data 
Element lOH) and remove each action coded as “Commercial Item Procedures not 
used.” 

Rationale: According to the FPDS-NG “User Manual” (2014), 

This data element is required for a DCA, Purchase Order, and populates 
on all orders and PBA Calls and Modifications. This designates whether 
the solicitation used the special requirements for the acquisition of 
commercial items or other supplies or services authorized to use 
commercial item procedures intended to more closely resemble those 
customarily used in the commercial marketplace as defined by FAR Part 
12. Select the appropriate value from the drop down menu. See Data 
Dictionary Element lOH Use Case for appropriate data entry requirements. 

(p. 55) 

FAR Subpart 13.5, Test Program for Certain Commercial Items, is only 
applicable to commercial items. Removing all actions coded as “Commercial Item 
Procedures not used” removes all actions for non-commercial items that are not eligible 
for the test program. 


Step 6: Filter Column AT (Commercial Item Test Program—Data Element lOJ) 
and remove cells with “No.” 

Rationale: According to the FPDS-NG “User Manual” (2014), 

This data element is required for a DCA, Purchase Order, and populates 
on all orders and PBA Calls and Modifications. This field designates 
whether the acquisition utilized FAR 13.5 Test Program for Certain 
Commercial Items. The FAR 13.5 Test Program provides for the use of 
simplified acquisition procedures for the acquisition of supplies or 
services in amounts greater than the simplified acquisition threshold when: 
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(1) the acquisition does not exceed the threshold and the contracting 
officer reasonably expects that offers will only include commercial items; 

(2) the acquisition does not exceed the threshold and is for commercial 
items that, as determined by the head of the agency, are to be used in 
support of a contingency operation or to facilitate the defense against or 
recovery from nuclear, biological, chemical, or radiological attack; or (3) 
the acquisition does not exceed the threshold and can be treated as an 
acquisition of commercial items in accordance with FAR 12.102(f)(1)— 
agency head determines the supplies or services are to be used to facilitate 
defense against or recovery from nuclear, biological, chemical, or 
radiological attack, (p. 56) 

This final sort shows the total actions eligible to use FAR Subpart 13.5, Test 
Program for Certain Commercial Items, versus how many actions actually used the 
program. 

C. METHODOLOGY LIMITATIONS 

Three main limitations were identified during the course this research. The first is 
an inherent limitation due to the data relying on human data entry and self-reporting. As 
discussed in the literature review, accuracy and reliability issues have been found in 
FPDS-NG reporting, although actions coded as having utilized the test program were 
accurate (GAO, 2014). 

The second limitation was discovered through performing data analysis and stems 
from a discrepancy between two data fields. Data cell Column AJ (Extent Competed— 
Data Element lOA) “is the code that represents the competitive nature of the contract” 
(EPDS-NG, 2014, p. 47). Table 6 illustrates the various options available: 


34 



Table 6. Options for FPDS-NG Data Element lOA—Extent Competed 

(from EPDS-NG, 2014, p. 47) 


Short Description 

Full Description 

Competed under SAP 

Select this code when the action is 
competed under the Simplified 

Acquisition Threshold. 

Pollow On to Competed Action. This 
value is not available for Post CEOD 
actions. 

Select this code when the action is a 
follow on to an existing competed 
contract. PAR 6.302-1 

Pull and Open Competition 

Report this code if the action resulted 
from an award pursuant to PAR 6.102(a) - 
sealed bid, PAR 6.102(b) - competitive 
proposal, PAR 6.102(c) - combination, or 
any other competitive method that did not 
exclude sources of any type 

Pull and Open Competition after exclusion 
of sources 

Select this code when some sources are 
excluded before competition 

Not Available for Competition 

Select this code when the contract is not 
available for competition 

Not Competed 

Select this code when the contract is not 
competed. 

Not Competed under SAP 

Select this code when the action is NOT 
competed under the Simplified 

Acquisition Threshold 


Eor actions coded as utilizing the PAR 13.5 Test Program under Data Element 
lOJ, all actions were rightfully coded as either “Competed under SAP” or “Not Competed 
under SAP;” however, there were also actions in excess of $150,000, but less than 
$6,500,000 coded as “Competed under SAP” or “Not Competed under SAP” that were 
not coded as utilizing the PAR 13.5 Test Program. Generally, this situation should not 
exist because the authority for utilizing SAP in excess of $150,000 is limited to the test 
program, which authorizes the use up to $6,500,000, or declared contingencies, which 
authorize the use up to $300,000 inside the United States and up to $1,000,000 outside 
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the United States (FAR 2.101, 2014). Of those actions that were coded as either 
“Competed under SAP” or “Not Competed under SAP,” 349 actions were less than or 
equal to $1,000,000, leaving a potential for SAP usage in support of a contingency, with 
49 additional actions that appear to at least be miscoded in one way or another. 
Unfortunately, FPDS-NG data does not contain a data field to distinguish between 
actions in support of a contingency and those that are not, which is the third and final 
limitation. 

The inability to identify contingency contracting actions limits the accuracy of the 
methodology in two ways. First, since the SAT is higher for declared contingencies, not 
all actions in excess of $150,000 require the authority of FAR 13.5 to use SAP. This 
could result in the methodology somewhat overstating the test program’s full usage rate; 
however, the overseas contingency contracts issued in Afghanistan use an Army 
Department of Defense Activity Address Code (DODAAC) because the Army is the lead 
agency in the theater of operations. Since those contracts would not appear in an Air 
Force data set pulled from FPDS-NG, the risk of overstating the program’s usage rate is 
minimal. Second, the FAR 13.5 Test Program’s upper threshold for contingencies is 
$12,000,000. As a result, this methodology may understate the full usage rate since any 
actions in support of a contingency in excess of $6,500,000 were not counted. Of all 
contracts between $6,500,000 and $12,000,000, only three actions were coded as using 
the test program. While FPDS-NG does contain a mandatory data field to indicate if a 
contract action “supports a declared contingency operation, a declared humanitarian 
operation, or a declared peacekeeping operation” (Contingency, Humanitarian, or 
Peacekeeping Operation—Data Element 6H), that field is not part of the FPDS-NG 
competition report (FPDS-NG, 2014, p. 36). If that field was present in the data, both of 
the previous limitations could be corrected for through additional filters and sorting to 
specifically identify and factor in those additional test program actions. 

Of these limitations, using an alternate final step in the methodology can assess 
the second issue concerning the discrepancy in coding: 
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Alternate Step 6: Filter Column AJ (Extent Competed—Data Element lOA) and 
keep cells with “Competed Under SAP” and “Not Competed Under SAP.” 

This alternative final sort shows the total number of actions over $150,000 but 
less than or equal to $6,500,000 where Simplified Acquisition Procedures were used 
either competitively or noncompetitively out of the same pool of eligible contracts. 
Theoretically, this number should be identical to the number of actions coded as using the 
test program since FAR 13.5 is the authority for utilizing SAP in excess of the SAT. Any 
inconsistencies between the two computed values would signify miscoding in one of the 
two fields. 

D. CHAPTER SUMMARY 

The research methodology described in this section leverages the lessons obtained 
from the literature review to create a schema devised to improve on past FAR 13.5 test 
program studies. The result is a detailed step-by-step process designed to control for key 
factors, such as dollar thresholds and mandatory sources, and ultimately aims to capture 
the most accurate test program usage rate possible for the U.S. Air Force from February 
1,2013, to May 30, 2014. 

The three identified limitations, human data entry, FPDS-NG data field 
discrepancy, and contingency contracting actions, each provide a unique challenge. 
However, the design schema was developed to reduce the effects of the limitations as 
much as possible. Human data entry and self-reporting have and will continue to affect 
accuracy levels of FPDS-NG data. However, recent studies have shown accuracy of 
FPDS-NG data pertaining to the FAR 13.5 test program have improved significantly and 
are “sufficiently reliable to determine the minimum extent to which the test program was 
used by the components” (GAO, 2014, p. 27). Additionally, despite the improvements of 
the FPDS-NG contract reporting system, errors in the data fields still exist. To account 
for a data field error that specifically affects the FAR 13.5 test program data, an alternate 
step in the methodology was developed resulting in a usage range. Finally, while the data 
set does not enable the ability to adjust for contingency contract actions, avoiding U.S. 
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Army DODAACs significantly reduces the risk of overstating the program’s usage rate. 
Chapter IV presents the data resulting from the applied methodology and analyzes the 
results. 
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IV. DATA PRESENTATION AND ANALYSIS 


A. CHAPTER INTRODUCTION 

The purpose of this chapter is to analyze the Air Force’s usage rate of the FAR 
Subpart 13.5 Test Program for Certain Commercial Items. First, test program awards will 
be identified using the methodology described in Chapter III and the dollar value 
distribution examined. Level of competition and average degree of competition will then 
be provided and analyzed. Lastly, Air Force policy, guidance, and training will be 
discussed as possible conditions affecting the data results. 

B. ANALYZING THE AIR FORCE’S USAGE RATE OF THE TEST 

PROGRAM 

Utilizing the previously defined methodology in Chapter III produced the 
following results. 

I. Identifying Test Program Awards in the Air Force 

The FPDS-NG data from February 1, 2013, through May 31, 2014, contained a 
total of 156,248 contract actions. These contract actions consisted of any new contract 
awards, modifications, purchase and delivery orders, etc., that were reported through 
FPDS-NG during that period of time. Of all those contract actions, 18,995 were of a 
dollar value greater than the SAT ($150,000) but less than or equal to $6,500,000. Since 
those actions still contained modifications, factoring those out resulted in a total of 
11,570 new contracts or purchase/delivery orders awarded within the test program’s 
range. Next, eliminating new actions utilizing ordering vehicles such as BPAs and ID/IQs 
reduced the number of actions to 3,508. As non-commercial acquisitions were removed, 
the total number of actions eligible for the test program within the data set was 1,709. 
Finally, filtering the remaining eligible contract actions returned only 475 actions 
reporting use of the test program, a usage rate of just 27.79%. Use of the alternate 
methodology, which filtered results based on the competition coding instead of the test 
program coding, on the same eligible pool of 1,709 contract actions returned 880 actions, 

a usage rate of 51.49%, that reported the use of SAP (either competitively or non- 
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competitively) in excess of $150,000, but less than or equal to $6,500,000. Figure 2 
graphically depicts the results of these filters. 

Step 1:156,248 Actions 

Step 2:18,995 Actions 

Step 3:11,570 Actions 

Step 4: 3,508 Actions 

Step 5:1,709 
Actions 

Step 6: 

475 to 880 
Actions 

Figure 2. Identifying the Air Foree’s Number of Test Program Awards 
between $150,000 and $6.5M (after FPDS-NG, n.d.) 

Sinee the Test Program for Certain Commercial Items authorizes the use of 
simplified procedures in exeess of the SAT, both methodologies should theoretically 
report the same result; however, only 475 actions were eoded as having utilized the test 
program while 880 actions (comprised of those same 475 plus another 405) were eoded 
as having utilized simplified procedures. Because of the inconsisteney in data fields that 
should be synonymous with one another, it would be presumptuous to eite one result as 
representing the true usage rate of the test program for the Air Force while ignoring the 
other or to assume how actions were miseoded without aceess to additional contraet 
doeumentation. Instead, the two data points , 27.79% and 51.49%, (as shown in Table 7) 
should represent the floor and eeiling, respeetively, for the use of test program provisions 
in Air Force. 
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Table 7. Air Force Test Program Awards Between $150,000 and $6.5M 

(after FPDS-NG, n.d.) 


METHODOLOGY 

TEST PROGRAM 
NUMBER 

TEST PROGRAM 
PERCENT 

Primary: Commercial Item 



Test Program - Data 

475 

27.79% 

Element lOJ 



Alternate: Extent Competed 

- Data Element lOA 

880 

51.49% 


Theoretically, since the FAR requires contracting activities to “employ the 
simplified procedures authorized by the test to the maximum extent practicable,” the 
number of actions coded as utilizing the Test Program for Certain Commercial Items 
should ideally be at or near the number of eligible actions of 1,709 (FAR 13.500(b)); 
however, in the case of both methodologies, a substantial number of new contract actions 
failed to use the test program. Since IDV’s were factored out of the eligible pool under 
this methodology, it stands to reason that any non-test program actions had to utilize the 
more complex, exhaustive procedures required by FAR Part 14, Sealed Bidding, or FAR 
Part 15, Contracting by Negotiation. Considering that the CAO (2014) reported “the test 
program contracts [they] reviewed were generally awarded more quickly and with less 
administrative burden than had the contracts been awarded using negotiated procedures,” 
(p. 14) Air Force contracting officials are seemingly missing out on opportunities to 
reduce the time and resources spent on at least 48.51% and up to 72.21% of commercial 
requirements (CAO, 2014). 
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2. 


Distribution of Air Force Test Program Awards 


Another area that was analyzed was the distribution of test program usage based 
on the dollar amount of the requirement. Using the original methodology’s Commercial 
Item Test Program—Data Element lOJ, dollar values were separated into two categories, 
test program and non-test program actions, then segmented in $500,000 increments. 
Figure 3 graphically portrays the results. 
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Figure 3. Test Program and Non-Test Program Actions by Dollar Value 

(after FPDS-NG, n.d.) 


Overall, there were only minor deviations from the overall average usage rate of 

27.79% with only two sets of values exceeding a difference of 6%. Those two deviations 

were acquisitions between $3,500,000 and $4,000,000, which totaled 48 observations 
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with a usage rate of only 16.67% and acquisitions between $5,500,000 and $6,000,000, 
which totaled only 10 observations with a usage rate of 50.00%. In general, this data 
appears to indicate that any perceived risk associated with higher dollar value 
acquisitions does not significantly impact the rate at which the test program is used by 
Air Force contracting officials. 

C. TEST PROGRAM COMPETITION STATISTICS 

In addition to containing data that provided the usage rate of the Test Program for 
Certain Commercial Items, the FPDS-NG data included additional information that could 
potentially illustrate the effectiveness of the program. One foundational component of the 
Federal Acquisition System is competition. Required by CICA (unless otherwise 
exempted), competition is integral toward ensuring efficiency, effectiveness, and 
transparency in government contracts (Manuel, 2011). The GAO reaffirms this notion in 
their report on competition in federal contracting by stating “The benefits of competition 
in acquiring goods and services from the private sector are well established. Competitive 
contracts can help save the taxpayer money, improve contractor performance, curb fraud, 
and promote accountability for results” (GAO, 2010, p. 1). Based on the important role of 
competition in contracting, it simply made sense to use the available data to assess the 
test program’s ability to fulfill this public policy objective and compare the results to 
non-test program and IDV actions. 

I. Level of Competition 

The same FPDS-NG report contained data on whether an action was competitive 
and even the amount of competition that was obtained. Column AJ of the excel 
spreadsheet contained the “Extent Competed—Data Element lOA” and column AS 
contained the “Number of Offers Received—Data Element lOD” (EPDS-NG, 2014). 
Multiple filters were used to separate test program, non-test program, and IDV actions. 
Next, the competition information from both data elements was used to calculate how 
many actions were competed, how many actions were competed with only one offer 
received, and how many actions were not competed. Eigure 4 depicts the calculations and 
corresponding results. 
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Figure 4. Percentage of Competitive, Noncompetitive, and Competed 
Contracts Actions with Only One Offer Received 
(after FPDS-NG, n.d.) 

The figure shows that test program actions had the highest rate of competition at 
75.79% while also minimizing the instances in which only one offer is received at 
14.44%. On the other hand, non-test program actions for commercial items over the same 
dollar range had the lowest rate of competition at 53.16%. Meanwhile, IDVs, an 
increasingly popular acquisition technique, had the highest incident of competition with 
only one offeror at 20.13% (it is worth noting that this methodology controlled for 
multiple award IDVs to determine this rate). 
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2. Average Degree of Competition 

Column AS, “Number of Offers Received - Data Element lOD,” was also used to 
calculate the average number of offerors participating in response to the various 
acquisition methods. Again, the same filters were used to separate test program, non-test 
program, and IDV actions from one another before calculating the average. Figure 5 
displays those results for comparison. 
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Figure 5. Average Number of Offers Received (after FPDS-NG, n.d.) 



According to the FPDS-NG data reports, more offerors generally responded to 
requirements solicited using test program procedures than either non-test program or IDV 
procedures. While the raw numerical differences between the various methods appear 
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inconsequential, the computed numbers are small to begin with so the difference is best 
represented as a percentage. On average, for those actions that were competed, test 
program solicitations received 5.60% more offerors than non-test program solicitations 
and 11.03% more offerors than those utilizing IDVs (the number of observations for test 
program, non-test program, and IDV actions were 475, 1,234, and 2,325, respectively). 

3. Section Summary 

Ultimately, since correlation does not prove causation, these results do not 
necessarily indicate that more actions are competed or more offers are received as a result 
of utilizing test program procedures; however, the data does at least affirm that the use of 
the test program’s simplified and streamlined procedures has not circumvented 
competition requirements. Furthermore, the significant correlation of increased 
competition and test program usage seems to warrant an investigation into the causal 
relationship between acquisition methods and levels of competition. 

D. TEST PROGRAM POLICY, GUIDANCE, AND TRAINING 

In an effort to identify what factors may be limiting the test program’s usage, this 
paper will examine the related policy, guidance, and training to determine whether they 
have a positive, negative, or neutral impact. 

I. Policy 

The Test Program for Certain Commercial Items has been in existence for 
approximately 18 years since its original passage under FARA; however, during that 
time, the program has always remained temporary, requiring reauthorization by Congress 
each time it expired. The result is increased unpredictability for contracting officials that 
is further complicated by periods where the test program has outright expired without 
being renewed, like on January 1, 2012. In that instance, the authority for the program 
lapsed until over a year later, on January 15, 2013, when the NDAA for FY 2013 was 
passed to temporarily extend the authority for the test program until January 1, 2015 
(DPAP, 2013). The memorandum used to inform the DOD acquisition workforce of the 
program’s extension further illustrates the volatile nature of the program when it stated. 
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“This class deviation is effective until it is incorporated in the FAR or is otherwise 
rescinded” (DPAP, 2013, p. 1). In other words, contracting professionals interested in 
using the test program during the time period while the FAR was not updated needed to 
further verify that another memorandum was not published rescinding the authority of the 
first memorandum, which authorized the temporary deviation. Consider another situation 
in which a contracting professional references an older (pre-2012) hard copy of the FAR 
and mistakenly determines that the test program is no longer valid because it lists the 
January 1, 2012 expiration date. Ultimately, the tenuous situation does not help the usage 
rate of the test program, nor does it inspire confidence for contracting officials to devote 
time, effort, and resources toward developing guidance or training on the use of test 
program procedures and how to best take advantage of them. 

2. Guidance 

Guidance on test program procedures is scarce. Other than the information 
contained in FAR Subpart 13.5, the DFARS and AFFARS are completely silent when it 
comes to the test program. Furthermore, the Air Force Installation Contracting Agency 
(AFICA) Mandatory Procedures (MP), which governs operational contracting squadrons 
(the agencies in the Air Force most likely to use the test program because they acquire 
primarily commercial items), also makes no mention of the test program. Therefore, 
beyond the requirement at FAR 13.500(b) to “employ the simplified procedures 
authorized by the test to the maximum extent practicable,” the DoD and AF do nothing 
further to encourage or require its use. 

3. Training 

From a training perspective, the mission of the Defense Acquisition University 
(DAU) is to “Provide a global learning environment to develop qualified acquisition, 
requirements and contingency professionals who deliver and sustain effective and 
affordable warfighting capabilities” (DAU, n.d.a, para. 1). The DAU is the primary 
organization for DOD agencies, including the Air Force, to obtain contracting related 
training and information. Per icatalog.dau.mil, the DAU’s offerings are broken down into 
two categories: training courses and Continuous Learning Modules (CLM). The DAU 
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currently offers 23 different contracting courses with one course, CON 237, addressing 
Simplified Acquisition Procedures (DAU, n.d.c). Figure 6 displays DAU’s learning 
objectives for CON 237. 



DEFENSE ACQUISITION UNIVERSITY 

CON 237 - Simplied Acquisition Procedures 

090210 

Course Learning/Performance Objectives followed by its 
enabling learning objectives on separate lines if specified. 


1 

Given a description of a requirement, identify the benefits of using Simpiified Acquisition Procedures (SAP). 

Recognize the benefits of using SAP. 

Identify the regulatory and legal basis for SAP. 

Define the Simplified Acquisition Threshold (SAT). 

2 

Given a description of a requirement, determine if Simpiified Acquisition Procedures (SAP) can be used. 

Determine if a requirement can be satisfied using simplified acquisition procedures. 

Determine if a requirement can be satisfied by purchasing a commercial or non-developmental item. 

Describe the advantages of purchasing commercial items. 

3 

Given a description of a requirement, identify potentiai market research resources and techniques. 

Describe the reasons to conduct market research. 

Identify market research techniques. 

Identify market research resources. 

4 

Given a description of a requirement, determine when required sources and mandatory items must be used. 

Determine when required sources and mandatory items must be used. 

Identify the requirements for small business set-asides when using SAP. 

Describe the information tracked from SAP source lists. 

5 

Given a description of a requirement, select a purchase method to be used in conjunction with the Simplified Acquisition Procedures (SAP). 

Determine when to use the Governmentwide commercial purchase card. 

Determine when it is practicable to use FACNET/EDI. 

Describe the advantages of using blanket purchase agreements. 

Identify situations when written purchase orders are required. 

Determine when to use other payment methods (SF 44/imprest fund/third party draft). 

6 

Given a description of a requirement, select a solicitation method to be used in conjunction with the Simplified Acquisition Procedures (SAP). 

Select the solicitation method to be used based on the requirement. 

Identify the public announcement requirements associated with each solicitation method. 

Determine synopsis requirements associated with each solicitation method. 

Describe the competition requirements associated with each solicitation method. 

7 

Given a description of a requirement, determine the types of evaluation factors to use in Simplified Acquisition Procedures (SAP) solicitations. 

Select a quote based on price-related factors. 

Determine when to use non-price-related factors. 

Select the types of non-price-related factors to include in a solicitation based on requirements description. 

8 

Given a description of a post-award issue, identify the actions that should be taken according to FAR Part 13. 

Describe the quality assurance limitations when purchasing commercial items. 

Describe the importance of warranties in purchasing commercial items. 

Identify when purchase orders should be terminated or canceled. 

Determine when options can be exercised in SAP. 

9 

Given a statement describing the major points covered in the preceding lessons on Simplified Acquisition Procedures, select the corresponding 
definition, policy, regulation, threshold, or procedure. 


Page 1 of 1 

Figure 6. DAU Learning Objectives for CON 237—Simplified Acquisition 

Procedures (from DAU, n.d.a) 

Despite an entire course devoted to FAR Part 13 and Simplified Acquisition 
Procedures, none of the course’s learning/performance objectives make any mention of 
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the Test Program for Certain Commercial Items. Likewise, the brief description of the 
CLM covering Simplified Acquisition Procedures, CLC 005, does not reference the test 
program either when it merely states “This Simplified Acquisition Procedures Overview 
(SAP) Continuous Learning Module (CLM) is designed to provide Federal procurement 
professionals with a better understanding of contracting for suppliers and services using 
SAP” (DAU, 2014, para. 1). 

4. Section Summary 

Ultimately, these three conditions do not create an environment conducive to the 
use of test program procedures. The lack of a consistent policy means the authority for 
the program is unpredictable, which could increase buyers’ apprehension about 
employing the procedures and negatively impact the test program’s usage. Furthermore, 
the temporary nature of the test program is quite possibly a driving or contributing factor 
to the vast lack of training and educational materials relating to its use, as agencies may 
be reluctant to invest resources in developing curricula and informative materials for a 
process that may not exist in a few years. These lack of materials mean individuals are 
not exposed to the existence of the test program, how it functions, or its merits, which 
would further limit its usage. Taken together, these circumstances limit the contracting 
professionals in the Air Force from employing simplified procedures in excess of the 
SAT, resulting in a usage rate that is less than optimal. 

E. CHAPTER SUMMARY 

This chapter analyzed the Air Force’s usage rate of the FAR Subpart 13.5 Test 
Program for Certain Commercial Items through filtering data in the FPDS-NG 
Competition Report. The chapter also identified the various competition metrics and 
compared them to alternate acquisition procedures. Lastly, areas where policy, guidance, 
and training are lacking were discussed as factors potentially limiting the use of test 
program procedures. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. CHAPTER INTRODUCTION 

The purpose of this chapter is to summarize the findings of this research paper. 
The conclusion will summarize the highlights of the test program, answer the research 
paper’s two research questions, then provide recommendations to support further 
improvements in the usage of the test program. This chapter will also highlight a few 
areas for expanding research on the test program. 

B. CONCLUSIONS 

As explained in detail in Chapter II, Section B, the test program procedures 
authorized by FAR Subpart 13.5 are designed to provide a number of benefits to 
acquisition professionals contracting for commercial items between $150,000 and 
$6,500,000. Those benefits are: 

1. Reduced administrative costs by limiting file documentation and 
authorizing a more efficient acquisition process. 

2. Improved opportunities for socio-economic business concerns with 
shorter, simpler solicitations that are more easily understood and by 
allowing any interested companies to respond versus IDVs, which are 
more exclusive. 

3. The promotion of efficiency and economy in contracting through a 
streamlined evaluation process and the ability to use innovative 
approaches, such as reverse auctions. 

4. Avoiding unnecessary burdens for agencies and contractors by allowing 
the use of RFQs, which are generally less burdensome while still 
permitting the use of RFPs and IFBs if the situations warrants. 

Generally, if a process is simpler and less time consuming, individuals should 
naturally be expected to embrace and utilize such a practice; however, despite being 
established to extend the benefits of SAP to a larger number of commercial item 
acquisitions, that does not appear to have occurred in regard to the Test Program for 
Certain Commercial Items. The results of this research culminated in the following 
answers to the research questions: 
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1. What was the Air Force’s usage rate of FAR 13.5 Test Program for Certain 

Commercial Items procedures from February 1, 2013, through May 31, 

2014? 

After factoring out alternate procurement methods (such as ID Vs) that may be 
either required or viewed by a contracting official as equally or more efficient, the usage 
rate of the test program in the Air Force is still lagging somewhere between 27.79% and 
51.49% of all eligible contract actions. In other words, approximately half to three- 
quarters of Air Force commercial acquisitions between $150,000 and $6,500,000 
needlessly subject themselves to more complex, exhaustive procedures in order to acquire 
goods and services that are predominantly available in the commercial market. 

2. What factors may be limiting the use of FAR 13.5 Commercial-Item Test 

Program procedures? 

Some likely contributors to the low usage rate of the test program in the Air Force 
were the following: 

• The temporary nature of the Test Program for Certain Commercial Items. 

• The absence of guidance within the DOD and AF to encourage or require 
the test program’s use. 

• The lack of training and educational materials on test program procedures. 

Another factor impacting the ability to definitively assess the usage rate of the test 
program within the Air Force was the inconsistency in coding contract actions in FPDS- 
NG. In order to alleviate or eliminate these issues, this paper makes the following 
recommendations. 

C. RECOMMENDATIONS 

1. Recommendation 1—Permanently Authorize the Test Program for 
Certain Commercial Items 

The Test Program for Certain Commercial Items has been in existence for 
approximately 18 years from the time it was originally enacted under FARA; however, 
during that time, the program has remained temporary, requiring reauthorization by 
Congress each time it expired. During the GAO’s research into the usage of the test 
program, the “DHS and DOI suggested that the temporary nature of the test program 
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hindered its use and recommended that the authority be made permanent” (GAO, 2014, 
p. 23). 

When this research began, the initial version of the NDAA for FY 2015, H.R. 
4435, dated May 13, 2014, did not contain a provision to extend the authority of the test 
program; however. Congressman Gerry Connolly, representing the 11th district of 
Virginia, proposed an amendment “to make the authority to use simplified acquisition 
procedures for certain commercial items permanent” (House of Representatives, 
Committee on Rules, 2014, Arndt. 204. As of September 29, 2014, the current version of 
the NDAA for FY 2015, dated June 5, 2014, which was on the Senate’s calendar for 
consideration, would, if passed by the Senate and signed by the President, make the Test 
Program for Certain Commercial Items permanent by amending “Section 4202 of the 
Clinger-Cohen Act of 1996 (division D of Public Law 104-106; 10 U.S.C. 2304 note)” to 
strike subsection (e) (NDAA for FY 2015, 2014, p. 277). The results of this research 
strongly support this change to the test program in order to eliminate the previous 
tenuous, unpredictable status of its authority. 

An additional potential benefit of making the test program permanent was 
identified when 

the Office of Management and Budget noted that in the event that the 
authority was made permanent, it would expect agencies to periodically 
review their use of the test program as part of their management reviews. 
Consequently, we believe that collecting and assessing data on the 
program’s use would be beneficial. (GAO, 2014, p. 23) 

2. Recommendation 2—Incorporate FAR 13.5 Usage Provisions Into the 
AFICA MP 

The Air Force Installation Contracting Agency (AFICA) should incorporate FAR 
13.5 usage provisions into their mandatory procedures to encourage or require the use of 
the test program for eligible acquisitions. This requirement would not supersede other 
mandatory source or disallow the use of an available IDV if, in the judgment of the 
contracting professional, an alternate method was the better procurement option; 
however, those 48.51% to 72.21% of contract actions that use more complex, exhaustive 
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procedures with no justifiable reason should be reviewed and required to use the more 
efficient and economical approach provided by the test program. 

AFICA’s MP states that their organization is responsible for reviewing all 
contract actions in excess of $2,000,000 (AFICA, 2013). During the period analyzed by 
this research, 189 contract actions valued at $2,000,000 or higher did not use test 
program procedures. If AFICA had required those organizations to use the test program 
on those requirements as part of their review that would have single-handedly increased 
the usage rate by over 10%. Moreover, it is plausible that these procedures, if enforced by 
AFICA on acquisitions $2,000,000 and higher, would trickle down within agencies to 
eligible actions of lower dollar amounts, further increasing the Air Force’s usage rate. 

3. Recommendation 3—Incorporate FAR 13.5 Into DAU SAP Training 

Even if Congress makes the Test Program for Certain Commercial Items 
permanently available and AFICA requires the program’s usage, contracting 
professionals must understand the procedures in order to correctly and effectively employ 
them. Therefore, the DAU should incorporate the test program methodology and 
procedures into its CON 237 “Simplified Acquisition Procedures” training module and 
create an additional Continuous Learning Module specifically for the use of SAP in 
excess of the SAT as authorized by FAR 13.5. 

4. Recommendation 4—Update FPDS-NG to Prevent Coding 
Inconsistencies 

In order to improve the accuracy of test program data, FPDS-NG should be 
updated by the General Services Administration to correct the current coding 
inconsistency identified by this research between Data Element lOJ “Commercial Item 
Test Program” and Data Element lOA “Extent Competed.” Currently, the system 
prevents actions coded under Data Element lOJ as having utilized the test program from 
choosing any competitive process besides “competed under SAP” and “not competed 
under SAP.” Conversely, the update should prevent contract action reports (CARs) from 
finalizing if a non-contingency action in excess of the SAT is coded under Data Element 
lOA as either “competed under SAP” or “not competed under SAP,” but not coded as 
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using the test program under Data Element lOJ (since the authority to use SAP in excess 
of $150,000 comes from the test program). 


D. AREAS FOR FURTHER INVESTIGATION AND RESEARCH 

This research can be expanded and improved upon in several ways. The following 
list includes areas for additional research opportunities: 

1. Researchers could examine random samples of eligible contract files to 
identify any potential issues with under or over reporting. Physically 
examining contract files would also be useful to identify any best practices 
used by contracting professionals in employing the authority of the test 
program. 

2. Researchers could survey and interview operational contracting officials to 
obtain qualitative data on the test program. The survey should focus on 
randomly sampling acquisition professionals in operational contracting 
squadrons since those acquisitions are most likely to be eligible for the test 
program (i.e., commercial items between $150,000 and $6,500,000). The 
survey should seek to obtain information that assesses contracting 
professionals’ awareness, understanding, and application of test program 
procedures. 

3. Researchers could survey and interview contractors to obtain qualitative 
data on the test program from an industry perspective. The survey should 
focus on assessing their preference and competence when it comes to 
solicitations under FAR 13.5 versus more lengthy and complex 
solicitations under FAR Part 14 or 15. 

E. CHAPTER SUMMARY 

This chapter concluded the research into the Air Force’s usage rate of the FAR 
Subpart 13.5 Test Program for Certain Commercial Items. Multiple recommendations 
were made to improve the usage of the test program or the ability to analyze that usage. 
Finally, future researchers curious about the test program’s effectiveness and benefits 
were provided with a few areas worthy of additional investigation. 
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